Skip to content

separateDebugInfo: add symlinks to executable and source for debuginfod support#394674

Merged
symphorien merged 10 commits intoNixOS:stagingfrom
symphorien:separatedebuginfo_debuginfod_support
Jun 20, 2025
Merged

separateDebugInfo: add symlinks to executable and source for debuginfod support#394674
symphorien merged 10 commits intoNixOS:stagingfrom
symphorien:separatedebuginfo_debuginfod_support

Conversation

@symphorien
Copy link
Member

For debuginfod support we must be able to map a build-id to

  • debug symbols
  • the original elf file for which the debug symbols where separated
  • the corresponding source files

Currently, hydra provides an index from build-id to the nar of the debug output containing the debug symbols.

Add symlinks in these outputs so that we can recover the store path of the source and original elf file. We can then fetch them by the normal binary cache protocol.

Pros:

  • we could host an official instance of such a debuginfod server

Cons

A proof of concept of such debuginfo server is available at https://github.com/symphorien/nixseparatedebuginfod2

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@nix-owners nix-owners bot requested a review from Ericson2314 March 30, 2025 17:05
@symphorien symphorien changed the base branch from master to staging March 30, 2025 17:07
@github-actions github-actions bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 8.has: documentation This PR adds or changes documentation labels Mar 30, 2025
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 30, 2025
@symphorien
Copy link
Member Author

Currently this only allows serving unpatched source. If source files are patched, debuginfod servers can only serve the unpatched source.
Alternatives:

  • systematically copy the source after patchPhase to the $debug output (or another output)
  • detect files modified during patch phase, and only copy those to the $debug output. Somewhat more complex.

@github-actions github-actions bot added 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog This PR adds or changes release notes 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: vim Advanced text editor 6.topic: ocaml OCaml is a general-purpose, high-level, multi-paradigm programming language. 6.topic: TeX Issues regarding texlive and TeX in general 6.topic: vscode A free and versatile code editor that supports almost every major programming language. 6.topic: games Gaming on NixOS 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 6.topic: policy discussion Discuss policies to work in and around Nixpkgs backport release-24.11 labels Mar 30, 2025
@nix-owners
Copy link

nix-owners bot commented Mar 30, 2025

The PR's base branch is set to staging, but 89 commits from the master branch are included. Make sure you know the right base branch for your changes, then:

  • If the changes should go to the master branch, change the base branch to master
  • If the changes should go to the staging branch, rebase your PR onto the merge base with the staging branch:
    # git rebase --onto $(git merge-base upstream/staging HEAD) $(git merge-base upstream/master HEAD)
    git rebase --onto c68937a0a0008af9117305874bd3da814fac1f19 10c515901f74f353957a311035639c0b30cf8bf5
    git push --force-with-lease

@symphorien symphorien force-pushed the separatedebuginfo_debuginfod_support branch from 59845de to dfbb727 Compare March 30, 2025 22:11
@github-actions github-actions bot removed 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog This PR adds or changes release notes 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: vim Advanced text editor 6.topic: ocaml OCaml is a general-purpose, high-level, multi-paradigm programming language. 6.topic: TeX Issues regarding texlive and TeX in general labels Mar 30, 2025
@github-actions github-actions bot added the 6.topic: COSMIC COSMIC is a software platform for designing beautiful user experiences label May 29, 2025
@symphorien symphorien marked this pull request as draft May 29, 2025 08:34
@symphorien symphorien force-pushed the separatedebuginfo_debuginfod_support branch from 77f7676 to 28cf81f Compare May 29, 2025 09:12
@github-actions github-actions bot removed 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: qt/kde Object-oriented framework for GUI creation 6.topic: kernel The Linux kernel 8.has: changelog This PR adds or changes release notes 8.has: module (update) This PR changes an existing module in `nixos/` labels May 29, 2025
@symphorien
Copy link
Member Author

I added logic to copy patched files to the debug output so that the debuginfod server can serve them.

the symlink to the source creates new references, and copying patched files might create more. This is thus incompatible with allowedReferences et at. I added some logic that disables these attributes but for the debug output only. this required structured attrs. With grep I determined that only a dozen derivations are affected in nixpkgs.

I also encountered issues with autopatchelfhook, see the commit message for the exact issue. I taught autopatchelfhook to ignore the debug output.

@wolfgangwalther
Copy link
Contributor

Re-running CI due to a odd failure fixed in #416448

Copy link
Member

@mweinelt mweinelt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks and sounds plausible. I think we should give it a try.

Please fix the scoping of the commit message for the commit that adds the release notes entry.

…od support

A debuginfod support must be able to map a build-id to
- debug symbols
- the original elf file for which the debug symbols where separated
- the corresponding source files

Currently, hydra provides an index from build-id to the nar of the debug
output containing the debug symbols.

Add symlinks in these outputs so that we can recover the store path of
the source and original elf file. We can then fetch them by the normal
binary cache protocol.

About source files: to minimize storage demands, in the ideal case,
software would be built from the source store path $src and the
debuginfod server would just have to serve source files from this store
path. In practice, source files are sometimes patched as part of the
build. This commit stores the modified files in the debug output is a so
called source overlay so that the debuginfod serve can serve the patched
content of the file.

The checksum was chosen as follows (where big is 4GB of zeros):

$  hyperfine -L s sysv,bsd,crc,sha1,sha224,sha256,sha384,sha512,blake2b,sm3 'cksum -a {s} big'
Benchmark 1: cksum -a sysv big
  Time (mean ± σ):     854.5 ms ± 270.5 ms    [User: 245.3 ms, System: 601.8 ms]
  Range (min … max):   760.5 ms … 1623.8 ms    10 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (1.624 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark 2: cksum -a bsd big
  Time (mean ± σ):      5.838 s ±  0.045 s    [User: 5.118 s, System: 0.693 s]
  Range (min … max):    5.767 s …  5.897 s    10 runs

Benchmark 3: cksum -a crc big
  Time (mean ± σ):     829.9 ms ±  28.6 ms    [User: 274.5 ms, System: 551.0 ms]
  Range (min … max):   803.2 ms … 904.8 ms    10 runs

Benchmark 4: cksum -a sha1 big
  Time (mean ± σ):      2.553 s ±  0.010 s    [User: 1.912 s, System: 0.631 s]
  Range (min … max):    2.543 s …  2.575 s    10 runs

Benchmark 5: cksum -a sha224 big
  Time (mean ± σ):      2.716 s ±  0.018 s    [User: 2.054 s, System: 0.645 s]
  Range (min … max):    2.695 s …  2.743 s    10 runs

Benchmark 6: cksum -a sha256 big
  Time (mean ± σ):      2.751 s ±  0.029 s    [User: 2.057 s, System: 0.674 s]
  Range (min … max):    2.712 s …  2.812 s    10 runs

Benchmark 7: cksum -a sha384 big
  Time (mean ± σ):      5.600 s ±  0.049 s    [User: 4.820 s, System: 0.753 s]
  Range (min … max):    5.515 s …  5.683 s    10 runs

Benchmark 8: cksum -a sha512 big
  Time (mean ± σ):      5.543 s ±  0.021 s    [User: 4.751 s, System: 0.768 s]
  Range (min … max):    5.523 s …  5.579 s    10 runs

Benchmark 9: cksum -a blake2b big
  Time (mean ± σ):      5.091 s ±  0.025 s    [User: 4.306 s, System: 0.764 s]
  Range (min … max):    5.048 s …  5.125 s    10 runs

Benchmark 10: cksum -a sm3 big
  Time (mean ± σ):     14.220 s ±  0.120 s    [User: 13.376 s, System: 0.783 s]
  Range (min … max):   14.077 s … 14.497 s    10 runs

Summary
  cksum -a crc big ran
    1.03 ± 0.33 times faster than cksum -a sysv big
    3.08 ± 0.11 times faster than cksum -a sha1 big
    3.27 ± 0.11 times faster than cksum -a sha224 big
    3.31 ± 0.12 times faster than cksum -a sha256 big
    6.13 ± 0.21 times faster than cksum -a blake2b big
    6.68 ± 0.23 times faster than cksum -a sha512 big
    6.75 ± 0.24 times faster than cksum -a sha384 big
    7.03 ± 0.25 times faster than cksum -a bsd big
   17.13 ± 0.61 times faster than cksum -a sm3 big

unfortunately, crc (and sysv) are not supported by --check, so they are
disqualified. sha1 sha224 and sha256 are sensibly as fast as one
another, so let's use a non broken one, even though cryptographic
qualities are not needed here.
needed for separateDebugInfo
needed for separateDebugInfo = true
needed by separateDebugInfo = true
neeeded by separateDebugInfo = true
needed by separateDebugInfo = true
… by separateDebugInfo

this output contains unpredictable references by design. It would nearly
always break allowedReferences et al. As it's not intended to enter a
real closure, but to be fetched on demand by buildid, disable reference
whitelisting instead, but only for this output. This requires
__structuredAttrs. Only a dozen derivations are affected in the whole of
nixpkgs.
required for separateDebugInfo
otherwise systemd fails to build because autoPatchelfHook patches libs
with debug objects and this creates circular dependencies from out to
debug (of course, debug depends on out)

example of faulty behavior:

searching for dependencies of /nix/store/c8cd7b82py02f0rkags351nhg82wwjm6-systemd-minimal-257.5/bin/systemd-delta
    libsystemd-shared-257.so -> found: /nix/store/icpqrawjhsw4fbi4i2hp7cxvf3kbzq7m-systemd-minimal-257.5-debug/lib/debug
    libc.so.6 -> found: /nix/store/7lyblga0bzjk0sl93vp4aiwbvb57vp2x-glibc-2.40-66/lib
@jys1670
Copy link
Contributor

jys1670 commented Jul 15, 2025

Did this break jetbrain jdk?
It now fails with BOOT_JDK: unbound variable (hydra)
But BOOT_JDK is clearly defined:

EDIT: yep, setting

  separateDebugInfo = false;
  __structuredAttrs = false;

fixes jetbrains.jdk build for me

@symphorien
Copy link
Member Author

I'm currently building this, but this should fix it: #425529

@lf-
Copy link
Member

lf- commented Jul 16, 2025

FYI this caused some regressions in Lix that are quite hard for us to fix because overrideAttrs can't touch the inputs to this check: https://git.lix.systems/lix-project/lix/issues/917#issuecomment-13264

Now, we could just backport it and roll some releases, but that's kind of annoying, so I'm just going to point out that this has some annoying out-of-tree effects here in case there's a desire to fix it another way :)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/unstable-fails-to-build-stable-ok-flakes/67837/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: java Including JDK, tooling, other languages, other VMs 6.topic: policy discussion Discuss policies to work in and around Nixpkgs 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: stdenv Standard environment 6.topic: systemd Software suite that provides an array of system components for Linux operating systems. 8.has: documentation This PR adds or changes documentation 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants